Lexically Constrained Knowledge Distillation for Neural Machine Translation

Knowledge distillation is a representative approach in neural machine translation (NMT) for compressing large model into lightweight one. This first trains strong teacher model, and then forces more compact student to imitate the teacher. Although key successful knowledge constructing stronger using state-of-the-art NMT may remain inadequate owing errors. Accordingly, an severely degrades due error propagation, especially regarding words important sentence meaning. To mitigate degradation problem, we propose method lexical constraint as privileged information NMT. The proposed with constraint, list of automatically extracted from target training data. We configure according importance fallibility Models trained our result improved compared those baseline English↔German English↔Japanese tasks under condition without ensemble decoding beam-search decoding.